test: automated testing prototype — pytest infrastructure + unit tests for 5 integrations#250
Merged
TheRealAgentK merged 22 commits intomasterfrom Apr 20, 2026
Merged
Conversation
🔍 Integration Validation ResultsCommit: Changed directories:
|
TheRealAgentK
force-pushed
the
test/automated-testing-prototype
branch
from
71ddc0b to
ef9aa11
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
TheRealAgentK
force-pushed
the
test/automated-testing-prototype
branch
from
27c8bf9 to
9cf3eb0
Compare
- Add pytest/coverage config (pyproject.toml, requirements-test.txt) - Add root conftest.py with shared fixtures and Integration.load() patch - Add .env.example for integration test credentials - Add unit tests for hackernews (29), bitly (37), nzbn (41), notion (39), shopify-customer (31) - Update CONTRIBUTING.md with test setup, running, and coverage docs - Add TEST_STOCKTAKE.md with full audit of existing test landscape All 177 tests pass with zero credentials required. Coverage: bitly 92%, hackernews 89%, nzbn 76%, notion 67%, shopify-customer 60%.
- Remove unused variable assignments (F841) - Add nosec B105 for test credential strings - Add noqa E402 for imports after sys.path.insert - Remove unused imports (F401) - Apply ruff format
CI runs ruff with the tooling's ruff.toml config which uses line-length=120. Previous format commit used the default line-length=88, causing mismatches. This reformats all files in the 5 integration directories plus conftest.py using the correct config.
…rror tests The SDK may either raise ValidationError (output schema validation) or return a result with error indicators, depending on the installed version. Added _expect_validation_error() helper that accepts both behaviors.
…alidation errors Pin autohive-integrations-sdk~=1.0.2 in requirements-test.txt to match what shopify-customer (and all other integrations) actually use. SDK 1.0.2 raises ValidationError on output schema mismatches without catching it internally, so tests must use pytest.raises(ValidationError). Removed the _expect_validation_error helper that masked this.
…idation SDK version is owned by each integration's requirements.txt, not the test infrastructure. The tooling's run_tests.py now installs deps per-integration. Tests use pytest.raises(ValidationError) which matches SDK 1.0.2 behavior (raises instead of catching internally).
TheRealAgentK
force-pushed
the
test/automated-testing-prototype
branch
from
cd5c812 to
b972487
Compare
…eanup Ported the Perplexity search integration from the private integrations repo. Replaced hardcoded API key with PERPLEXITY_API_KEY environment variable lookup. Returns a clear error message if the env var is not set or empty. Updated README and error messages to reference the env var. No secrets committed. Version bumped to 1.1.0 (minor) for the auth change.
- 29 mocked unit tests for search_web action (all passing) - 10 e2e integration tests (skip if PERPLEXITY_API_KEY not set) - tests/conftest.py for sys.path setup - .env.example updated with PERPLEXITY_API_KEY placeholder
Superseded by test_perplexity_unit.py (29 mocked tests) and test_perplexity_integration.py (10 e2e tests). All 8 original scenarios are covered plus 20 additional cases.
- 12 e2e tests covering all 8 actions against the live HN API - No credentials needed (public API) - Removes test_hackernews.py (manual asyncio.run runner) and context.py
- 10 e2e tests covering all 9 read-only actions against the live Bitly API - Skip if BITLY_ACCESS_TOKEN not set; 4 tests also skip if no bitlinks exist - Write actions (shorten_url, create_bitlink, update_bitlink) excluded - Removes test_bitly.py (manual asyncio.run runner) and context.py
…d pyproject.toml - Add comparison table showing file naming, markers, discovery, and CI behavior - Fix incorrect integration test command (pytest -m integration dir/ doesn't work) - Add integration test template and writing guide - Add clarifying comments to pyproject.toml
Collaborator
Author
|
Note: The structural test failures in this PR are currently expected and will go away with the next tooling bump (already PR'ed)
|
TheRealAgentK
requested review from
NinosMan,
ProRedCat,
Shubhank-Jonnada and
sumitramanga
ProRedCat
approved these changes
Apr 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces automated testing infrastructure for Autohive integrations, including pytest configuration, unit tests for 6 integrations, and end-to-end integration tests for 3 of those.
What's in this PR
1. Pytest Infrastructure
pyproject.tomlasyncio_mode=auto,python_files=test_*_unit.py,-m unitdefault, coverage settingsconftest.py(root)Integration.load()for test discovery + stdlib.envloaderrequirements-test.txtpytest,pytest-asyncio,pytest-cov— no SDK (each integration provides its own).env.example.envholds actual values).gitignore.env2. Unit Tests (206 tests, all CI-safe)
Mocked tests that run without credentials or network access. Auto-discovered by pytest via
test_*_unit.pynaming.Total: 198 unit tests (auto-discovered by CI)
Note: shopify-customer has known output schema bugs — error paths return
Nonefor fields declared astype: "object". SDK 1.0.2 raisesValidationError; tests usepytest.raises(ValidationError). 4 actions (update_profile, update_address, delete_address, refresh_token) lack mocked test coverage.3. Integration Tests (32 e2e tests, never run in CI)
End-to-end tests that call real APIs. Excluded from CI by two mechanisms:
python_filesrestricts auto-discovery totest_*_unit.pyaddoptsincludes-m unit, which deselects@pytest.mark.integrationtestsTests skip gracefully via
pytest.skip()if required credentials are not set.BITLY_ACCESS_TOKENPERPLEXITY_API_KEYTotal: 32 integration tests (local-only, credential-gated)
Run locally:
4. Perplexity Integration (new)
Ported from the private
Autohive-AI/integrationsrepo with security cleanup:os.environ.get("PERPLEXITY_API_KEY")+ error handlingREADME.md5. Legacy Test Cleanup
Removed old manual test runners (
asyncio.run(main())scripts) and theircontext.pyfiles for integrations that now have proper pytest coverage:test_hackernews.py,context.pytest_hackernews_unit.py(29) +test_hackernews_integration.py(12)test_bitly.py,context.pytest_bitly_unit.py(37) +test_bitly_integration.py(10)test_perplexity.py,context.pytest_perplexity_unit.py(29) +test_perplexity_integration.py(10)nzbn, notion, and shopify-customer still have their legacy test files (removal pending).
6. Documentation
CONTRIBUTING.md: New "Running Tests" section with comparison table, correct commands for both test types, templates for writing unit and integration testspyproject.toml: Comments explainingpython_filesrestriction and how to run integration testsTEST_STOCKTAKE.md: Audit of all 86 integrations' test coverage, patterns, and gaps7. CI-related fixes
ruffcompliance (tooling'sruff.toml, line-length=120)banditsecurity fixes (nosec annotations where appropriate)Test Architecture
Key Design Decisions
requirements-test.txt: Each integration pins its own SDK version inrequirements.txt. The CI test runner (run_tests.pyin tooling) installs deps per-integration to respect version pins.python_files = ["test_*_unit.py"]: Integration tests are excluded from discovery at the file level, not just the marker level. This prevents accidental CI execution even if someone forgets the marker.test_*_integration.pynot matched bypython_files) AND marker (-m unitinaddoptsdeselects@pytest.mark.integration). Belt and suspenders.Integration.load()monkeypatch: The SDK resolvesconfig.jsonrelative to its own package location, which breaks when installed as a site-package. The rootconftest.pymonkeypatches it to use frame inspection. A proper fix (passing explicit paths) is documented inTEST_STOCKTAKE.md.Tooling Repo Changes (separate PRs)
This work was supported by changes to autohive-integrations-tooling:
run_tests.py,action.ymlupdates)run_tests.pygit diff -w)run_tests.pydocumentationconftest.pyas alternative tocontext.pyin validation, document unit/integration test separation